Skip processed artifacts when reprocessing a run - #200
Merged
Conversation
Processed posters and videos often match watcher filename patterns, so run reprocess queued them and left them stuck in processing. Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Skipping the filename gate on Function URL invokes was meant to keep a user-initiated reprocess from being silently dropped, but processors return without touching the file row when the name isn't theirs, so the web app's `processing` transition was left with nothing to resolve it. The gate now always runs and a mismatch is PATCHed to `failed`, matching the existing recovery for a missing instrument or unmapped type. Also derive `ReprocessableFile` from the `files` row so a mistyped category or status literal fails to compile. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
Clicking Reprocess on a run queued every file on it — including the posters, videos and CSVs that Data Hub generated itself. The Lambda only knows how to read original instrument files, so it looked at those, decided they weren't its job, and quietly did nothing. The files were left sitting in
processingforever with nothing to move them along.DishCam hit this hardest: its posters are
.jpg, which is also an extension the watcher picks up as raw data.What changes
Only original files can be reprocessed. Generated files are skipped by run-level Reprocess, rejected with a clear error by the API and the MCP tool, and no longer show a Reprocess button.
When the Lambda can't handle a file, it now says so. A manual reprocess used to skip the filename check entirely, so the file reached a processor that ignored it and left it stuck. Now an unrecognised file is marked
failedwith a reason you can read on the row — same as when the instrument is missing or has no processor. This closes the same bug for original files with unexpected names, like a stray.txton a DishCam run.Not fixed here
Files already stuck in
processingfrom earlier reprocesses stay stuck — the app won't reprocess a file in that state. They need a one-offPATCH /api/v1/files/:idback touploaded.Test plan
canReprocessFilerejects generated files, allows raw onesMade with Cursor